Tip: Quoted sections like this will provide helpful instructions on how to navigate and use a Jupyter notebook.
Over the past decade, bicycle-sharing systems have been growing in number and popularity in cities across the world. Bicycle-sharing systems allow users to rent bicycles for short trips, typically 30 minutes or less. Thanks to the rise in information technologies, it is easy for a user of the system to access a dock within the system to unlock or return bicycles. These technologies also provide a wealth of data that can be used to explore how these bike-sharing systems are used.
In this project, you will perform an exploratory analysis on data provided by Motivate, a bike-share system provider for many major cities in the United States. You will compare the system usage between three large cities: New York City, Chicago, and Washington, DC. You will also see if there are any differences within each system for those users that are registered, regular users and those users that are short-term, casual users.
Before looking at the bike sharing data, you should start by asking questions you might want to understand about the bike share data. Consider, for example, if you were working for Motivate. What kinds of information would you want to know about in order to make smarter business decisions? If you were a user of the bike-share service, what factors might influence how you would want to use the service?
Question 1: Write at least two questions related to bike sharing that you think could be answered by data.
Answer: Bike sharing data can talk about various varibles which can affect the business like which stations are popular, location is normally near to office or residential area, age and demography. Are there repeat customers or they drop off after 1st time.
Tip: If you double click on this cell, you will see the text change so that all of the formatting is removed. This allows you to edit this block of text. This block of text is written using Markdown, which is a way to format text using headers, links, italics, and many other options using a plain-text syntax. You will also use Markdown later in the Nanodegree program. Use Shift + Enter or Shift + Return to run the cell and show its rendered form.
Now it's time to collect and explore our data. In this project, we will focus on the record of individual trips taken in 2016 from our selected cities: New York City, Chicago, and Washington, DC. Each of these cities has a page where we can freely download the trip data.:
If you visit these pages, you will notice that each city has a different way of delivering its data. Chicago updates with new data twice a year, Washington DC is quarterly, and New York City is monthly. However, you do not need to download the data yourself. The data has already been collected for you in the /data/ folder of the project files. While the original data for 2016 is spread among multiple files for each city, the files in the /data/ folder collect all of the trip data for the year into one file per city. Some data wrangling of inconsistencies in timestamp format within each city has already been performed for you. In addition, a random 2% sample of the original data is taken to make the exploration more manageable.
Question 2: However, there is still a lot of data for us to investigate, so it's a good idea to start off by looking at one entry from each of the cities we're going to analyze. Run the first code cell below to load some packages and functions that you'll be using in your analysis. Then, complete the second code cell to print out the first trip recorded from each of the cities (the second line of each data file).
Tip: You can run a code cell like you formatted Markdown cells above by clicking on the cell and using the keyboard shortcut Shift + Enter or Shift + Return. Alternatively, a code cell can be executed using the Play button in the toolbar after selecting it. While the cell is running, you will see an asterisk in the message to the left of the cell, i.e.
In [*]:. The asterisk will change into a number to show that execution has completed, e.g.In [1]. If there is output, it will show up asOut [1]:, with an appropriate number to match the "In" number.
## import all necessary packages and functions.
import csv # read and write csv files
from datetime import datetime # operations to parse dates
from pprint import pprint # use to print data structures like dictionaries in
# a nicer way than the base print function.
import csv
def print_first_point(filename):
"""
This function prints and returns the first data point (second row) from
a csv file that includes a header row.
"""
# print city name for reference
city = filename.split('-')[0].split('/')[-1]
print('\nCity: {}'.format(city))
with open(filename, 'r') as f_in:
## Use the csv library to set up a DictReader object. ##
## see https://docs.python.org/3/library/csv.html ##
trip_reader = csv.DictReader(f_in)
first_trip = next(trip_reader)
#for row in trip_reader:
## Use a function on the DictReader object to read the ##
## first trip from the data file and store it in a variable. ##
## see https://docs.python.org/3/library/csv.html#reader-objects ##
#first_trip = row
#print(
## Use the pprint library to print the first trip. ##
## see https://docs.python.org/3/library/pprint.html ##
# output city name and first trip for later testing
return (city, first_trip)
# list of files for each city
data_files = ['./data/NYC-CitiBike-2016.csv',
'./data/Chicago-Divvy-2016.csv',
'./data/Washington-CapitalBikeshare-2016.csv',]
# print the first trip from each file, store in dictionary
example_trips = {}
for data_file in data_files:
city, first_trip = print_first_point(data_file)
example_trips[city] = first_trip
print(example_trips[city])
If everything has been filled out correctly, you should see below the printout of each city name (which has been parsed from the data file name) that the first trip has been parsed in the form of a dictionary. When you set up a DictReader object, the first row of the data file is normally interpreted as column names. Every other row in the data file will use those column names as keys, as a dictionary is generated for each row.
This will be useful since we can refer to quantities by an easily-understandable label instead of just a numeric index. For example, if we have a trip stored in the variable row, then we would rather get the trip duration from row['duration'] instead of row[0].
It should also be observable from the above printout that each city provides different information. Even where the information is the same, the column names and formats are sometimes different. To make things as simple as possible when we get to the actual exploration, we should trim and clean the data. Cleaning the data makes sure that the data formats across the cities are consistent, while trimming focuses only on the parts of the data we are most interested in to make the exploration easier to work with.
You will generate new data files with five values of interest for each trip: trip duration, starting month, starting hour, day of the week, and user type. Each of these may require additional wrangling depending on the city:
datetime package will be very useful here to make the needed conversions.Question 3a: Complete the helper functions in the code cells below to address each of the cleaning tasks described above.
def duration_in_mins(datum, city):
"""
Takes as input a dictionary containing info about a single trip (datum) and
its origin city (city) and returns the trip duration in units of minutes.
Remember that Washington is in terms of milliseconds while Chicago and NYC
are in terms of seconds.
HINT: The csv module reads in all of the data as strings, including numeric
values. You will need a function to convert the strings into an appropriate
numeric type when making your transformations.
see https://docs.python.org/3/library/functions.html
"""
# Split the code as per the city
if city == "NYC" or city == "Chicago":
duration = float(datum['tripduration'])/60
print(duration)
elif city == "Washington":
duration = float(datum['Duration (ms)'])/(60*1000)
print(duration)
return duration
# Some tests to check that your code works. There should be no output if all of
# the assertions pass. The `example_trips` dictionary was obtained from when
# you printed the first trip from each of the original data files.
tests = {'NYC': 13.9833,
'Chicago': 15.4333,
'Washington': 7.1231}
for city in tests:
assert abs(duration_in_mins(example_trips[city], city) - tests[city]) < .001
%pdb
def time_of_trip(datum, city):
"""
Takes as input a dictionary containing info about a single trip (datum) and
its origin city (city) and returns the month, hour, and day of the week in
which the trip was made.
Remember that NYC includes seconds, while Washington and Chicago do not.
HINT: You should use the datetime module to parse the original date
strings into a format that is useful for extracting the desired information.
see https://docs.python.org/3/library/datetime.html#strftime-and-strptime-behavior
"""
# Split the code as per the city
if city == 'NYC':
dt_obj = datetime.strptime(datum['starttime'], "%m/%d/%Y %H:%M:%S")
month = dt_obj.strftime('%m')
month = int(month)
hour = dt_obj.strftime('%H')
hour = int(hour)
day_of_week = dt_obj.strftime('%A')
elif city == 'Washington':
dt_obj = datetime.strptime(datum['Start date'], "%m/%d/%Y %H:%M")
month = dt_obj.strftime('%m')
month = int(month)
hour = dt_obj.strftime('%H')
hour = int(hour)
day_of_week = dt_obj.strftime('%A')
elif city == 'Chicago':
dt_obj = datetime.strptime(datum['starttime'], "%m/%d/%Y %H:%M")
month = dt_obj.strftime('%m')
month = int(month)
hour = dt_obj.strftime('%H')
hour = int(hour)
day_of_week = dt_obj.strftime('%A')
return (month, hour, day_of_week)
# Some tests to check that your code works. There should be no output if all of
# the assertions pass. The `example_trips` dictionary was obtained from when
# you printed the first trip from each of the original data files.
tests = {'NYC': (1, 0, 'Friday'),
'Chicago': (3, 23, 'Thursday'),
'Washington': (3, 22, 'Thursday')}
for city in tests:
print(time_of_trip(example_trips[city], city))
assert time_of_trip(example_trips[city], city) == tests[city]
def type_of_user(datum, city):
"""
Takes as input a dictionary containing info about a single trip (datum) and
its origin city (city) and returns the type of system user that made the
trip.
Remember that Washington has different category names compared to Chicago
and NYC.
"""
# Split the code as per the city
if city == 'NYC' or city == 'Chicago':
user_type = datum['usertype']
print(user_type)
elif city == 'Washington':
user_type = datum['Member Type']
if user_type == 'Registered':
user_type = 'Subscriber'
print(user_type)
elif user_type == 'Casual':
user_type = 'Customer'
print(user_type)
return user_type
# Some tests to check that your code works. There should be no output if all of
# the assertions pass. The `example_trips` dictionary was obtained from when
# you printed the first trip from each of the original data files.
tests = {'NYC': 'Customer',
'Chicago': 'Subscriber',
'Washington': 'Subscriber'}
for city in tests:
assert type_of_user(example_trips[city], city) == tests[city]
Question 3b: Now, use the helper functions you wrote above to create a condensed data file for each city consisting only of the data fields indicated above. In the /examples/ folder, you will see an example datafile from the Bay Area Bike Share before and after conversion. Make sure that your output is formatted to be consistent with the example file.
def condense_data(in_file, out_file, city):
"""
This function takes full data from the specified input file
and writes the condensed data to a specified output file. The city
argument determines how the input file will be parsed.
HINT: See the cell below to see how the arguments are structured!
"""
with open(out_file, 'w') as f_out, open(in_file, 'r') as f_in:
# set up csv DictWriter object - writer requires column names for the
# first row as the "fieldnames" argument
out_colnames = ['duration', 'month', 'hour', 'day_of_week', 'user_type']
trip_writer = csv.DictWriter(f_out, fieldnames = out_colnames)
trip_writer.writeheader()
## Set up csv DictReader object ##
trip_reader = csv.DictReader(f_in)
# collect data from and process each row
for row in trip_reader:
# set up a dictionary to hold the values for the cleaned and trimmed
# data point
new_point = {}
## Use the helper functions to get the cleaned data from ##
## the original data dictionaries. ##
## Note that the keys for the new_point dictionary should match ##
## the column names set in the DictWriter object above. ##
duration = duration_in_mins(row, city)
new_point['duration'] = duration
month, hour, day_of_week = time_of_trip(row, city)
new_point['month'] = month
new_point['hour'] = hour
new_point['day_of_week'] = day_of_week
user_type = type_of_user(row, city)
new_point['user_type'] = user_type
## Write the processed information to the output file. ##
## see https://docs.python.org/3/library/csv.html#writer-objects ##
trip_writer.writerow({'duration': duration, 'month' : month, 'hour' : hour, 'day_of_week':day_of_week, 'user_type': user_type })
# Run this cell to check condense_data
city_info = {'Washington': {'in_file': './data/Washington-CapitalBikeshare-2016.csv',
'out_file': './data/Washington-2016-Summary.csv'},
'Chicago': {'in_file': './data/Chicago-Divvy-2016.csv',
'out_file': './data/Chicago-2016-Summary.csv'},
'NYC': {'in_file': './data/NYC-CitiBike-2016.csv',
'out_file': './data/NYC-2016-Summary.csv'}}
for city, filenames in city_info.items():
condense_data(filenames['in_file'], filenames['out_file'], city)
print_first_point(filenames['out_file'])